Establishing copy pairs from primary volumes to secondary volumes in multiple secondary storage systems for a failover session

ABSTRACT

Provided are a computer program product, system, and method for establishing copy pairs from primary volumes to secondary volumes in multiple secondary storage systems for a failover session. For each of the copy pairs, data is mirrored from the primary storage system to the associated secondary storage system in the copy pair. A failure is detected at the primary storage system. Selection is made of a selected secondary storage system of the secondary storage systems in response to detecting the failure, wherein a plurality of the secondary storage systems are available for selection. The selected secondary storage system is indicated as a new primary storage system to which host requests are directed.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a computer program product, system, andmethod for establishing copy pairs from primary volumes to secondaryvolumes in multiple secondary storage systems for a failover session.

2. Description of the Related Art

Failover programs, such as International Business Machine Corporation's(“IBM”) HyperSwap® which is a function in the z/OS® operating system,provides continuous availability for disk failures by maintainingsynchronous copies of all primary disk volumes on one or more primarystorage systems to one or more target (or secondary) storage systems.(HyperSwap is a registered trademark of IBM in countries throughout theworld). When a disk failure is detected, code in the operating systemidentifies HyperSwap managed volumes and instead of failing the I/Orequest, HyperSwap switches (or swaps) information in internal controlblocks so that the I/O request is driven against the secondary volume.Since the secondary volume is an identical copy of the primary volumeprior to the failure, the I/O request will succeed with no impact to theprogram issuing the I/O request, which could be an application programor part of the operating system. This therefore masks the disk failurefrom the program and avoids an application and/or system outage. Anevent which causes a HyperSwap to be initiated is called a “swaptrigger”.

Periodically HyperSwap checks the environment including the I/Oconfiguration to assure that no changes have occurred such that aHyperSwap is no longer possible. If there exists a problem in theconfiguration HyperSwap will indicate that it is either running indegraded mode or it may become completely disabled. In either case, if aHyperSwap occurred after the problem was detected some or all of thesystems in the sysplex (or computing cluster) may not be able toHyperSwap. When this happens the programs accessing the effected deviceswill fail which often means the entire system or sysplex will fail.

When a swap trigger is detected, HyperSwap first verifies that thetarget configuration is still viable by running the check one last time.If the configuration is still viable then the HyperSwap proceeds.Otherwise, if invalid, depending on policy, the HyperSwap is eitherterminated and backed out if required or systems that cannot proceedwith the HyperSwap are partitioned out of the sysplex in an attempt toallow other systems in the sysplex to complete the HyperSwap.

SUMMARY

Provided are a computer program product, system, and method forestablishing copy pairs from primary volumes to secondary volumes inmultiple secondary storage systems for a failover session. For each ofthe copy pairs, data is mirrored from the primary storage system to theassociated secondary storage system in the copy pair. A failure isdetected at the primary storage system. Selection is made of a selectedsecondary storage system of the secondary storage systems in response todetecting the failure, wherein a plurality of the secondary storagesystems are available for selection. The selected secondary storagesystem is indicated as a new primary storage system to which hostrequests are directed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a storage replication environment.

FIG. 2 illustrates an embodiment of an instance of a role pair.

FIG. 3 illustrates an embodiment of operations to establish a role pair.

FIG. 4 illustrates an embodiment of operations to establish a failoversession.

FIG. 5 illustrates an embodiment of operations to perform a failover.

FIG. 6 illustrates an embodiment of operations to validateconfigurations in a failover session.

FIG. 7 illustrates an embodiment of operations to add a storage systemto a failover session.

FIG. 8 illustrates a computing environment in which the components ofFIG. 1 may be implemented.

DETAILED DESCRIPTION

Described embodiments provide techniques for mirroring primary volumesin a primary storage system to multiple secondary storage systems, sothat each primary volume in the primary storage system is copied tomultiple secondary storage systems to be available if there is a failureat the primary storage system to increase the likelihood of surviving aprimary storage failure. A failover preference may specify a preferenceordering of the secondary storage systems to use as the new primarystorage system in the event the current primary storage system fails.The other available secondary storage systems not selected to become theprimary storage system may then provide secondary storage for the newprimary storage system.

FIG. 1 illustrates an embodiment of a mirror copy storage environmenthaving a host system 100 that is connected to a primary storage system102 and a plurality of secondary storage systems 104 ₁, 104 ₂, 104 ₃ towhich data from the primary storage system 102 is mirrored. The primarystorage system 102 includes a plurality of volumes 106 ₁, 106 ₂, 106 ₃,and 106 _(n) that are in copy pairs with corresponding volumes 108 ₁,108 ₂, 108 ₃, and 108 _(n), 110 ₁, 110 ₂, 110 ₃, and 110 _(n), and 112₁, 112 ₂, 112 ₃, and 112 _(n) in the secondary storage systems 104 ₁,104 ₂, 104 ₃. Host systems 100, 150 may access data in the currentlyindicated primary volumes for customer use and access. The hosts 100,150, primary storage system 102 and secondary storage systems 104 ₁, 104₂, 104 ₃ may communicate over a network 114.

The host system 100 includes a replication manager 116 to establishmirror copy relationships identified in role pairs 200. Each of the rolepairs 200 includes a plurality of copy pairs, each copy pair associatingone of the primary volumes 106 ₁, 106 ₂, 106 ₃, and 106 _(n) in theprimary storage system 102 with one of the secondary volumes 108 ₁, 108₂, 108 ₃, and 108 _(n), 110 ₁, 110 ₂, 110 ₃, and 110 _(n), and 112 ₁,112 ₂, 112 ₃, and 112 _(n) in one of the secondary storage systems 104₁, 104 ₂, 104 ₃, respectively. Thus, a role pair comprises a pluralityof copy pairs of primary volumes to secondary volumes in one of thesecondary storage systems 104 ₁, 104 ₂, 104 ₃. If the secondary storagesystem 104 ₁, 104 ₂, 104 ₃ is comprised of a plurality of separatestorage sub-systems, then the role pair 200 would include copy pairsindicating secondary volumes in different storage sub-subsystems of thelarger secondary storage system component. In a further embodiment, thestorage sub-systems may comprise logically partitioned sub-system of aphysical storage system.

The role pairs 200 comprise copy pairs with replication paths 120 ₁,120₂, 120 ₃ . . . 122 _(n), 122 ₁,122 ₂, 122 ₃ . . . 122 _(n), and 124₁,124 ₂, 124 ₃ . . . 124 _(n) between the primary volumes 106 ₁, 106 ₂,106 ₃, and 106 _(n) and the secondary volumes 108 ₁, 108 ₂, 108 ₃, and108 _(n), 110 ₁, 110 ₂, 110 ₃, and 110 _(n), or 112 ₁, 112 ₂, 112 ₃, and112 _(n), respectively, showing where the data in the primary volumes ismirrored to the secondary volumes in one of the secondary storagesystems. There may be multiple role pairs 200 for a replication session140, one role pair for each of the secondary storage systems 104 ₁, 104₂, 104 ₃ to which the primary volumes are replicated. The replicationmanager 116 may provide the role pairs 200 for the secondary storagesystems 104 ₁, 104 ₂, 104 ₃ to which the primary volume data is mirroredto a failover manager 126 to use to implement a failover from theprimary storage system 102 to one of the secondary storage systems 104₁, 104 ₂, 104 ₃ in response to a failure event.

The failover manager 126 maintains failover session information 130,which includes the role pairs 200 from the replication manager 116;failover preferences 132 that may comprise a user selected preferenceorder for selecting one of the secondary storage systems 104 ₁, 104 ₂,104 ₃ to failover to from the primary storage system 102; and primary134 and secondary 136 information having information used to access theprimary 102 and secondary storage systems 104 ₁, 104 ₂, 104 ₃.

In one embodiment, the hosts 150 may each have an instance of thefailover manager 126 and maintain a copy of the same failover sessioninformation 130, so that each host could separately implement afailover. Further, the failover manager 126 on each host allows thehosts 100, 150 to coordinate with peers a failover. However, in certainembodiments only one host 100 maintains a replication manager 116.

In the embodiment of FIG. 1 three secondary storage systems 104 ₁, 104₂, 104 ₃ are shown. However, in different implementations there may beat least two secondary storage systems and more than three secondarystorage systems. Further, there may be more or fewer primary andsecondary volumes than shown. The user may select an order of thesecondary storage systems 104 ₁, 104 ₂, 104 ₃ in the failoverpreferences 132 based on proximity to the primary storage system 102, sothat those secondary storage systems 104 ₁, 104 ₂, 104 ₃ closest to theprimary storage system 102 have a relatively higher ordering in thefailover preferences 132. For instance, one secondary storage system maybe in the same building, another in the same campus or facility, and athird in the same metropolitan area, etc.

The volumes 106 ₁, 106 ₂, 106 ₃, and 106 _(n), 108 ₁, 108 ₂, 108 ₃, and108 _(n), 110 ₁, 110 ₂, 110 ₃, and 110 _(n), and 112 ₁, 112 ₂, 112 ₃,and 112 _(n) may represent volumes stored in one storage system 102, 104₁, 104 ₂, 104 ₃, respectively, comprising a storage server or controllerand one or more attached storage devices. In an alternative embodiment,the storage systems 102, 104 ₁, 104 ₂, 104 ₃ may each be comprised of aplurality of storage sub-systems, wherein each subsystem comprises aserver/controller and one or more attached storage devices, and whereinthe volumes 106 ₁,106 ₂, 106 ₃, and 106 _(n), 108 ₁, 108 ₂, 108 ₃, and108 _(n), 110 ₁, 110 ₂, 110 ₃, and 110 _(n), and 112 ₁, 112 ₂, 112 ₃,and 112 _(n) in one of the storage systems 102, 104 ₁, 104 ₂, 104 ₃ aredistributed among the different storage sub-systems. The storage systemsand storage sub-systems may be comprised of an enterprise storagecontroller/server suitable for managing access to attached storagedevices, such as the International Business Machine Corporation's(“IBM”) DS8000® storage system. (DS8000 is a registered trademark of IBMin countries throughout the world).

In further embodiments, the storage systems 102, 104 ₁, 104 ₂, 104 ₃ mayeach comprise multiple storage systems, so that the primary storagesystem 102 may mirror its volumes to the different storage systemscomprising each of the storage systems 104 ₁, 104 ₂, 104 ₃.

In one embodiment, the replication manager 116 comprises a program formanaging the mirroring of volumes across systems, such as the IBMmirroring programs Geographically Dispersed Parallel Sysplex® (GDPS)®,and Tivoli® Storage Productivity Center for Replication (TPC-R) thatdefine a replication session and copy pairs 200. Different types ofmirroring may be selected to copy the data, such as synchronousmirroring, asynchronous mirroring or point-in-time mirroring, orcombinations of multiple of these different mirroring types. Thefailover manager 126 may comprise a program suitable for handling thefailover of a primary storage system 102 to one of the secondary storagesystems 104 ₁, 104 ₂, 104 ₃, such as the IBM HyperSwap product whichestablishes failover sessions from the established copy pairs.(Geographically Dispersed Parallel Sysplex, GDPS, Tivoli, and HyperSwapare registered trademarks of IBM in countries throughout the world).

In alternative embodiments, the functionality described with respect tothe replication manager 116 and failover manager 126 may be implementedin a single storage manager program or in multiple different programmodules.

The network 114 may comprise a Storage Area Network (SAN), Local AreaNetwork (LAN), Intranet, the Internet, Wide Area Network (WAN),peer-to-peer network, wireless network, arbitrated loop network, etc.The volumes may be implemented in one or more storage devices, or anarray of storage devices configured as Just a Bunch of Disks (JBOD),Direct Access Storage Device (DASD), Redundant Array of IndependentDisks (RAID) array, virtualization device, tape storage, flash memory,etc. Then storage devices in which the volumes are implemented maycomprise hard disk drives, solid state storage device (SSD) comprised ofsolid state electronics, such as a EEPROM (Electrically ErasableProgrammable Read-Only Memory), flash memory, flash disk, Random AccessMemory (RAM) drive, storage-class memory (SCM), etc., magnetic storagedisk, optical disk, tape, etc.

FIG. 2 illustrates an embodiment of an instance of a role pair 200, ofthe role pairs 200 that identifies secondary volumes in a secondarystorage system 104 _(j) comprising one of the secondary storage systems104 ₁, 104 ₂, 104 ₃ to which the volumes of the primary storage system102 are copied. The role pair 200, specifies copy pairs 208 ₁ . . . 208_(n), where there is one copy pair for each primary volume 106 ₁, 106 ₂,106 ₃, and 106 _(n) and each of the secondary volumes 108 ₁, 108 ₂, 108₃, 108 _(n), 110 ₁, 110 ₂, 110 ₃, and 110 _(n), or 112 ₁, 112 ₂, 112 ₃,and 112 _(n) to which each of the primary volumes is copied. If thesecondary storage system 104 j is comprised of a plurality of storagesub-systems, i.e., controllers, servers, etc., then the copy pairs 208 ₁. . . 208 _(n) may specify the secondary storage sub-system in which thesecondary volume is included.

In one embodiment, the hosts 100, 150 may include a channel subsystem,which has sub-channels that provide the information needed to access theunderlying storage devices in which the volumes are configured. The host100, 150 operating systems may include Unit Control Blocks (UCBs) thatprovide a software representation of the underlying volumes, and pointto the sub-channel that has the information on how to access the deviceincluding the volume addressed by the UCB. When an application allocatesa volume, such as volumes 106 ₁, 106 ₂, 106 ₃, 106 _(n), 108 ₁, 108 ₂,108 ₃, 108 _(n), 110 ₁, 110 ₂, 110 ₃, and 110 _(n), or 112 ₁, 112 ₂, 112₃, and 112 _(n), a UCB may be allocated for the volume includinginformation on the sub-channel used to access the device in which thevolume is included. Applications use the UCBs to access the volume Whenperforming a failover or Hyperswap, the UCB contents of the primary andsecondary volumes being swapped are swapped so that the UCB for theprimary volume points to the secondary volume and the UCB for thesecondary volume points to the primary volume. If the failover or swapresulted from a failure at the primary volume, then the UCB for thesecondary volume may point to a primary volume that is not accessibleafter the swap. Thus, swapping the UCBs physical addressing information(i.e. subchannel number) from the primary to the secondary deviceredirects the application's I/O requests to the secondary device

FIG. 3 illustrates an embodiment of operations performed by thereplication manager 116 to establish the role pairs 200 for a failoversession 130. Upon initiating (at block 300) operations to establish rolepairs 200, the replication manager 116 receives (at block 302) customerinput defining a replication session 140, including a primary storagesystem 102 including primary volumes 106 ₁, 106 ₂, 106 ₃, and 106 _(n)to replicate to multiple secondary storage systems 104 ₁, 104 ₂, 104₃,including secondary volumes 108 ₁, 108 ₂, 108 ₃, and 108 _(n), 110 ₁,110 ₂, 110 ₃, and 110 _(n), and 112 ₁, 112 ₂, 112 ₃, and 112 _(n) forwhich to establish role pairs 200, and a session type. In oneembodiment, the session type may comprise synchronous type copyoperation or non-synchronous copy types, such as asynchronous,point-in-time, etc. When the customer has completed selecting all of theprimary and secondary volumes in the secondary storage systems 104 ₁,104 ₂, 104 ₃ to include in the role pairs 200 for the replicationsession, the customer may invoke the replication manager 116 toestablish the replication copies.

The replication manager 116 establishes (at block 304) paths 120 ₁,120₂, 120 ₃ . . . 122 _(n), 122 ₁,122 ₂, 122 ₃ . . . 122 _(n), and 124₁,124 ₂, 124 ₃ . . . 124 _(n) between each of the primary and secondaryvolumes in the secondary storage systems identified in the role pairs200. The replication manager 116 establishes (at block 306) one rolepair 200 for each of the secondary storage systems 104 ₁, 104 ₂, 104 ₃,including a plurality of copy pairs 208 ₁ . . . 208 _(n) associating theprimary volumes and the corresponding secondary volumes to which theprimary volume data is copied in the secondary storage system. Afterestablishing the role pairs 200, the storage controller 102 beginscopying (at block 308) using the user specified copy type, e.g.,synchronous, data from the primary volumes to the secondary volumesassociated in the created role pairs 200, so that the primary volumesare copied to corresponding secondary volumes in multiple of thesecondary storage systems 104 ₁, 104 ₂, 104 ₃ identified in the rolepairs 200.

In response to completing copying of data from the primary volumes tothe secondary volumes for the copy pairs 208 ₁ . . . 208 _(n) of therole pairs 200, i.e., a full duplex state, provide (at block 310) therole pairs 200 to the failover manager 126 to use to create a failoversession 130 to enable failover from the primary storage system 102 tothe secondary storage system 104, for the role pairs 200. Copying ofupdates from the primary volumes to the secondary volumes may continueafter the full duplex state is established.

FIG. 4 illustrates an embodiment of operations performed by the failovermanager 126 to establish a failover session 130 for the received rolepairs 200. Upon initiating (at block 400) the operation to establish thefailover session 130, the failover manager 126 receives (at block 402)failover preferences 132 indicating an ordering of the secondary storagesystems 104 ₁, 104 ₂, 104 ₃, in which they should be selected forfailover if the primary storage system 102 fails. The failover manager126 further receives (at block 404) the established role pairs 200, onefor each of the secondary storage systems 104 ₁, 104 ₂, 104 ₃, toinclude in the failover session 130. The configuration of the copy pairs208 ₁ . . . 208 _(n) of the role pairs 200 is validated (at block 406)by testing the availability and accessibility of all the volumes in thecopy pairs 208 ₁ . . . 208 _(n) of the role pair 200s. If (at block 408)the configuration is not valid, then the failover manager 126 notifies(at block 410) the replication manager 116 that the failover session isdisabled. If (at block 408) the configuration is valid, then failovermode is enabled (at block 412) for the failover session 130 and the copypairs 208 ₁ . . . 208 _(n) of the received role pairs 200. Thereplication manager 116 is notified (at block 414) that the failoversession 130 has been established.

FIG. 5 illustrates an embodiment of operations performed by the failovermanager 126 to handle a failover from the primary storage system 102 toone of the secondary storage systems 104 ₁, 104 ₂, 104 ₃. Upon (at block500) the failover manager 126 detecting a failure (or permanent I/Oerror) of a primary storage system 102 being managed in a failoversession 130, the failover manager 126 quiesces (at block 502)Input/Output (I/O) requests to the primary volumes 106 ₁, 106 ₂, 106 ₃,and 106 _(n) from the host 100 and other host systems 150. The failovermanager 126 determines (at block 506) one of the secondary storagesystems 104 ₁, 104 ₂, 104 ₃ that is accessible and available (e.g.,valid configuration) to all the hosts 100, 150 and having the highestindicated order of the failover preferences 132 of the availablesecondary storage systems 104 ₁, 104 ₂, 104 ₃. The failover manager 126may cycle through the failover preferences 132 starting from the mostpreferred secondary storage system 104 ₁, 104 ₂, 104 ₃ until a mostpreferred secondary storage system that is available to all hosts 100,150 is determined. If (at block 508) there is no secondary storagesystem 104 ₁, 104 ₂, 104 ₃ that is available and accessible to all hosts100, then a customer failure policy is initiated (at block 510). Thefailure policy may partition the host systems 100, 150 out of thesysplex that cannot access the selected secondary storage system inorder to “save” the other host systems 100, 150 or, alternatively, thecustomer failure policy may indicate to terminate and back-out thefailover session 130.

If (at block 508) there is a valid configuration for one of thepreferred secondary storage systems 104 ₁, 104 ₂, 104 ₃, then thefailover manager 126 selects (at block 512) the determined secondarystorage system 104 ₁, 104 ₂, 104 ₃ as the selected secondary storagesystem and indicates (at block 514) the selected secondary storagesystem as a new primary storage system to which host 100, 150 requestsare directed. This indication may be made by updating the primary 134and secondary 136 information. The failover manager 126 may indicate (atblock 516) each of the available non-selected secondary storage systemsas secondary storage systems, such as by updating the secondary storageinformation 400 for each of the non-selected secondary storage systemsthat will continue to function as secondary storages or targets of themirroring from the new primary.

The failover manager 126 may invalidate (at block 518) the current rolepairs 200 for the failover session 130 to replace with new establishedrole pairs 200 having copy pairs for the remaining secondary storagesystems. The failover manager 126 (or replication manager 116, which maybe called by the failover manager 126) establishes (at block 520) copypaths between the new primary volumes and the secondary volumes in theith secondary storage system. The failover manager 126 establishes (atblock 522) role pairs 200 including copy pairs 208, associating theprimary volumes of the new primary storage system and the correspondingsecondary volumes to which the primary volume data is copied in theremaining secondary storage systems. After reestablishing copy paths andthe role pairs 200 from the new primary storage system to the remainingsecondary storage systems, the failover manager 130 ends (at block 524)quiescing I/Os to the primary storage system and allows I/Os to proceeddirected to the new primary storage system.

FIG. 6 illustrates an embodiment of operations performed by the failovermanager 126 to periodically check the health of the secondary storagesystems available in the failover session 130. Upon (at block 600)periodically validating the configurations for the failover session 130,the failover manager 126 validates the configuration of the paths 114 tothe secondary volumes in the secondary storage systems, such as byissuing a trivial I/O request to the secondary volumes to determinetheir availability. If (at block 604) the configuration is not valid,i.e., all copy paths and secondary storage systems are not available,then a determination is made (at block 608) whether the configuration isdegraded. A degraded state occurs when one or more hosts 100, 150 cannotaccess one or more of the secondary volumes in one or more of thesecondary storage systems 104 ₁, 104 ₂, 104 ₃, while other of the hosts100, 150 can access all of the secondary storage system volumes. If (atblock 608) the configuration is usable, then the configuration which isalso not valid at this point is degraded. In such case, a message ispresented (at block 610) to the operator or update to the replicationmanager interface that the configuration has an issue that should beaddressed, which may prevent a successful failover. This message is toencourage the administrator to correct the issue before a failoveroccurs. In a further scenario, the degraded state may involve one ormore hosts 100, 150 unable to access volumes in only one of thesecondary storage systems 104 ₁, 104 ₂, 104 ₃. In such case, theadministrator may still be alerted even though failover couldsuccessfully occur for at least one of the other secondary storagesystems whose volumes are accessible to all of the hosts 100, 150. Ifupon a next monitoring operation or configuration check, the problem isfixed, then the alert may be cleared. Further, before a failoveroperation, a configuration check may be performed to clear the alertbefore the failover is performed.

If (at block 608) the configuration is not usable, at which point it isalso not valid, then the failover session 130 is disabled (at block612). Further, if the configuration is valid (from the yes branch ofblock 604), control ends. With the embodiment of FIG. 6, a failoversession is disabled when all systems in the sysplex fail validation (forwhatever reasons). A session is degraded as long as at least one systemcan access all secondary volumes. In this way, the session is disabledif no one system will survive and degrade if at least one systemsurvives.

FIG. 7 illustrates an embodiment of operations performed by the failovermanager 126 or replication manager 116 to add a new storage system as asecondary storage system to an existing failover session 130, which maycomprise a repaired primary or secondary storage previously removed fromthe failover session 130 or a new storage system. Upon receiving (atblock 700) indication of a storage system to add to the failover session130, the failover manager 130 may include (at block 702), automaticallyor in response to customer input, the added storage system in thefailover preferences 132. For instance, if the added storage system isthe repaired old primary storage system removed due to a failure, it maybe added back to the failover preferences 132 with a highest preferencein the order. The operations at blocks 306-312 in FIG. 3 may beperformed to establish the copy paths and copy pairs for the newsecondary storage system to add to a new roll pair 200 to provide to thefailover session 130. The operations to add the copy pairs for the addedstorage system may be performed by establishing a new role pair as shownin FIG. 3 and copying the primary data to establish the duplex state forthe added secondary storage system.

The described embodiments provide techniques to mirror primary volumesto secondary volumes at multiple remote target secondary storage systemsby maintaining copy pairs between the primary volumes and the secondaryvolumes in each of the secondary storage systems. Upon detecting afailover at the primary storage system, one of the secondary storagesystems may be selected to be the new primary and the remainingsecondary storage systems are then configured to be the new secondarystorage systems to the new primary storage system, comprising theselected secondary storage system. Described embodiments providemultiple levels of protection by avoiding a single point of failure suchas the case if there is only one secondary storage system. The describedembodiments increase the likelihood of the system surviving a primarystorage failure if the target secondary storage system is not available,thus providing greater system availability.

The reference characters used herein, such as i and n, are used hereinto denote a variable number of instances of an element, which mayrepresent the same or different values, and may represent the same ordifferent value when used with different or the same elements indifferent described instances.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The computational components of FIG. 1, including the hosts 100, 150,primary storage system 102, and secondary storage systems 104 ₁, 104 ₂,104 ₃ may be implemented in one or more computer systems, such as thecomputer system 802 shown in FIG. 8. Computer system/server 802 may bedescribed in the general context of computer system executableinstructions, such as program modules, being executed by a computersystem. Generally, program modules may include routines, programs,objects, components, logic, data structures, and so on that performparticular tasks or implement particular abstract data types. Computersystem/server 802 may be practiced in distributed cloud computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

As shown in FIG. 8, the computer system/server 802 is shown in the formof a general-purpose computing device. The components of computersystem/server 802 may include, but are not limited to, one or moreprocessors or processing units 804, a system memory 806, and a bus 808that couples various system components including system memory 806 toprocessor 804. Bus 808 represents one or more of any of several types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, andnot limitation, such architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnects (PCI) bus.

Computer system/server 802 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 802, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 806 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 810 and/or cachememory 812. Computer system/server 802 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 813 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 808 by one or more datamedia interfaces. As will be further depicted and described below,memory 806 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 814, having a set (at least one) of program modules 816,may be stored in memory 806 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. The components of the computer 802 may be implemented asprogram modules 816 which generally carry out the functions and/ormethodologies of embodiments of the invention as described herein. Thesystems of FIG. 1 may be implemented in one or more computer systems802, where if they are implemented in multiple computer systems 802,then the computer systems may communicate over a network.

Computer system/server 802 may also communicate with one or moreexternal devices 818 such as a keyboard, a pointing device, a display820, etc.; one or more devices that enable a user to interact withcomputer system/server 802; and/or any devices (e.g., network card,modem, etc.) that enable computer system/server 802 to communicate withone or more other computing devices. Such communication can occur viaInput/Output (I/O) interfaces 822. Still yet, computer system/server 802can communicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 824. As depicted, network adapter 824communicates with the other components of computer system/server 802 viabus 808. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 802. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims herein after appended.

What is claimed is:
 1. A computer program product for maintaining arelationship between a primary storage system and a plurality ofsecondary storage systems, wherein the computer program productcomprises a computer readable storage medium having program instructionsembodied therewith, the program instructions executable by a processorto cause operations, the operations comprising: establishing a pluralityof copy pairs, where each of the copy pairs associates the primarystorage system with one of the secondary storage systems; for each ofthe copy pairs, mirroring data from the primary storage system to theassociated secondary storage system in the copy pair; detecting afailure at the primary storage system; selecting a selected secondarystorage system of the secondary storage systems in response to detectingthe failure, wherein a plurality of the secondary storage systems areavailable for selection; and indicating the selected secondary storagesystem as a new primary storage system to which host requests aredirected.
 2. The computer program product of claim 1, wherein theoperations further comprise: maintaining an indicated order ofpreference of selecting the secondary storage systems, wherein selectingthe selected secondary storage system comprises determining one of thesecondary storage systems that is available to connected hosts and hasthe highest indicated order of preference of the available secondarystorage systems, wherein the determined secondary storage systemcomprises the selected secondary storage system.
 3. The computer programproduct of claim 1, wherein the operations further comprise:establishing at least one new copy pair, wherein each of the new copypairs associates the new primary storage system with one of thesecondary storage systems not selected.
 4. The computer program productof claim 3, wherein the establishing the at least one new copy paircomprises: determining at least one of the secondary storage systemsother than the selected secondary storage system that is available,wherein the relationship is established with respect to the availablesecondary storage systems other than the selected secondary storagesystem.
 5. The computer program product of claim 1, wherein themirroring of data from the primary storage system to each of thesecondary storage systems comprises synchronously copying modified datafrom the primary storage system to the secondary storage system, andwherein each of the copy pairs are established in response to theassociated secondary storage in the copy pair having a full copy of thedata at the primary storage system.
 6. The computer program product ofclaim 5, wherein a replication manager performs the operations ofestablishing the copy pairs, wherein the operations further comprise:sending, by the replication manager, information on the copy pairs to afailover manager in response to establishing the copy pairs, wherein thefailover manager performs the operations of detecting the failure at theprimary storage system and selecting the selected secondary storagesystem; validating, by the failover manager, a configuration of the copypairs by testing an availability of the primary and secondary storagesystems in the copy pairs; communicating, by the failover manager, tothe replication manager that a failover session is enabled in responseto validating the configuration; and communicating, by the failovermanager, to the replication manager that the failover session is in adisabled state in response to failing validation of the configuration.7. The computer program product of claim 6, wherein the operationsfurther comprise: periodically checking, by the failover manager,whether the configuration of the copy pairs is valid; determining thatone of the secondary storage systems includes storage devices in thecopy pair that are unavailable; indicating that the determined secondarystorage system is not part of the failover session.
 8. The computerprogram product of claim 1, wherein the operations further comprise:determining that the primary storage system that experienced the failureis available; and establishing a copy pair associating the new primarystorage system with the primary storage system that experienced thefailure as a new secondary storage system.
 9. The computer programproduct of claim 1, wherein each of the primary storage system and thesecondary storage systems include a plurality of volumes, wherein thecopy pairs for the primary storage system and one of the secondarystorage systems associates each of the volumes in the primary storagesystem with one of the volumes in the associated secondary storagesystem, wherein data in the volumes in the primary storage system ismirrored to the associated volumes in each of the secondary storagesystems.
 10. A system for maintaining a relationship between a primarystorage system and a plurality of secondary storage systems, comprising:a processor; and a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya processor to cause operations, the operations comprising: establishinga plurality of copy pairs, where each of the copy pairs associates theprimary storage system with one of the secondary storage systems; foreach of the copy pairs, mirroring data from the primary storage systemto the associated secondary storage system in the copy pair; detecting afailure at the primary storage system; selecting a selected secondarystorage system of the secondary storage systems in response to detectingthe failure, wherein a plurality of the secondary storage systems areavailable for selection; and indicating the selected secondary storagesystem as a new primary storage system to which host requests aredirected.
 11. The system of claim 10, wherein the operations furthercomprise: maintaining an indicated order of preference of selecting thesecondary storage systems, wherein selecting the selected secondarystorage system comprises determining one of the secondary storagesystems that is available to connected hosts and has the highestindicated order of preference of the available secondary storagesystems, wherein the determined secondary storage system comprises theselected secondary storage system.
 12. The system of claim 10, whereinthe operations further comprise: establishing at least one new copypair, wherein each of the new copy pairs associates the new primarystorage system with one of the secondary storage systems not selected.13. The system of claim 12, wherein the establishing the at least onenew copy pair comprises: determining at least one of the secondarystorage systems other than the selected secondary storage system that isavailable, wherein the relationship is established with respect to theavailable secondary storage systems other than the selected secondarystorage system.
 14. The system of claim 10, wherein the mirroring ofdata from the primary storage system to each of the secondary storagesystems comprises synchronously copying modified data from the primarystorage system to the secondary storage system, and wherein each of thecopy pairs are established in response to the associated secondarystorage in the copy pair having a full copy of the data at the primarystorage system.
 15. The system of claim 14, wherein a replicationmanager performs the operations of establishing the copy pairs, whereinthe operations further comprise: sending, by the replication manager,information on the copy pairs to a failover manager in response toestablishing the copy pairs, wherein the failover manager performs theoperations of detecting the failure at the primary storage system andselecting the selected secondary storage system; validating, by thefailover manager, a configuration of the copy pairs by testing anavailability of the primary and secondary storage systems in the copypairs; communicating, by the failover manager, to the replicationmanager that a failover session is enabled in response to validating theconfiguration; and communicating, by the failover manager, to thereplication manager that the failover session is in a disabled state inresponse to failing validation of the configuration.
 16. The system ofclaim 10, wherein each of the primary storage system and the secondarystorage systems include a plurality of volumes, wherein the copy pairsfor the primary storage system and one of the secondary storage systemsassociates each of the volumes in the primary storage system with one ofthe volumes in the associated secondary storage system, wherein data inthe volumes in the primary storage system is mirrored to the associatedvolumes in each of the secondary storage systems.
 17. A computer programproduct for maintaining a relationship between a primary storage systemand a plurality of secondary storage systems, wherein the computerprogram product comprises a computer readable storage medium havingprogram instructions embodied therewith, the program instructionsexecutable by a processor to cause operations, the operationscomprising: establishing a plurality of copy pairs, where each of thecopy pairs associates the primary storage system with one of thesecondary storage systems; for each of the copy pairs, mirroring datafrom the primary storage system to the associated secondary storagesystem in the copy pair; detecting a failure at the primary storagesystem; selecting a selected secondary storage system of the secondarystorage systems in response to detecting the failure, wherein aplurality of the secondary storage systems are available for selection;and indicating the selected secondary storage system as a new primarystorage system to which host requests are directed.
 18. The computerprogram product of claim 17, wherein the operations further comprise:maintaining an indicated order of preference of selecting the secondarystorage systems, wherein selecting the selected secondary storage systemcomprises determining one of the secondary storage systems that isavailable to connected hosts and has the highest indicated order ofpreference of the available secondary storage systems, wherein thedetermined secondary storage system comprises the selected secondarystorage system.
 19. The computer program product of claim 17, whereinthe operations further comprise: establishing at least one new copypair, wherein each of the new copy pairs associates the new primarystorage system with one of the secondary storage systems not selected.20. The computer program product of claim 19, wherein the establishingthe at least one new copy pair comprises: determining at least one ofthe secondary storage systems other than the selected secondary storagesystem that is available, wherein the relationship is established withrespect to the available secondary storage systems other than theselected secondary storage system.
 21. The computer program product ofclaim 17, wherein the mirroring of data from the primary storage systemto each of the secondary storage systems comprises synchronously copyingmodified data from the primary storage system to the secondary storagesystem, and wherein each of the copy pairs are established in responseto the associated secondary storage in the copy pair having a full copyof the data at the primary storage system.
 22. The computer programproduct of claim 21, wherein a replication manager performs theoperations of establishing the copy pairs, wherein the operationsfurther comprise: sending, by the replication manager, information onthe copy pairs to a failover manager in response to establishing thecopy pairs, wherein the failover manager performs the operations ofdetecting the failure at the primary storage system and selecting theselected secondary storage system; validating, by the failover manager,a configuration of the copy pairs by testing an availability of theprimary and secondary storage systems in the copy pairs; communicating,by the failover manager, to the replication manager that a failoversession is enabled in response to validating the configuration; andcommunicating, by the failover manager, to the replication manager thatthe failover session is in a disabled state in response to failingvalidation of the configuration.
 23. The computer program product ofclaim 17, wherein each of the primary storage system and the secondarystorage systems include a plurality of volumes, wherein the copy pairsfor the primary storage system and one of the secondary storage systemsassociates each of the volumes in the primary storage system with one ofthe volumes in the associated secondary storage system, wherein data inthe volumes in the primary storage system is mirrored to the associatedvolumes in each of the secondary storage systems.